Power supply for a networking device with orthogonal switch bars

ABSTRACT

Power supply for a networking device may be provided. The networking device may comprise a first plurality of switch bars each comprising a first switch type arranged parallel to one another and a second plurality of switch bars each comprising a second switch type arranged parallel to one another. The first plurality of switch bars and the second plurality of switch bars may be arranged orthogonally. A first plurality of power supplies may be fed by a first source. A second plurality of power supplies may be fed by a second source. Respective ones of a first portion of the first plurality of power supplies feed first respective pairs of the first plurality of switch bars and respective ones of a first portion of the second plurality of power supplies feed second respective pairs of the first plurality of switch bars. The first respective pairs of the first plurality of switch bars may be different from the second respective pairs of the first plurality of switch bars.

RELATED APPLICATION TECHNICAL FIELD

Under provisions of 35 U.S.C. § 119(e), Applicant claims the benefit ofU.S. Provisional Application No. 62/755,586 filed Nov. 5, 2018, which isincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates generally to network data centers such ashigh-density enterprise data centers.

BACKGROUND

Cloud computing is a model that allows access to a shared pool ofconfigurable computing resources. Cloud computing and storage solutionsprovide users and enterprises with various capabilities to store andprocess their data in third-party data centers. It shares resources toachieve coherence and economies of scale.

Cloud computing also focuses on maximizing the effectiveness of theshared resources. Cloud resources are usually not only shared bymultiple users, but are also dynamically reallocated per demand. Thiscan work for allocating resources to users. For example, a cloudcomputer facility that serves European users during European businesshours with a specific application (e.g., e-mail) may reallocate the sameresources to serve North American users during North American businesshours with a different application (e.g., a web server). This approachhelps maximize computing power use while reducing the overall resourcescost by using, for example, less power, less air conditioning, and lessrack space, to provide the required functions. With cloud computing,like other client-server architectures, multiple users can access asingle server to retrieve and update their data without purchasinglicenses for different applications.

Interconnecting these shared resources is an important challenge. Inconventional data centers, several layers of IP switches, and largenumbers of cabled Ethernet connections complete the interconnect betweenservers, storage engines, networking devices, appliances, and otherendpoints to enable the cloud to perform its required applicationservices. In large data centers, these switches and interconnects can bedifficult to install and configure, consume large physical volumes,energy and cooling resources, and are expensive. Multiple layers ofswitches including spine, leaf and top of rack, and techniques such asfat trees, which may consist of layers of richly interconnectedswitches, are often employed to address these challenges.

BRIEF DESCRIPTION OF THE FIGURES

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various embodiments of the presentdisclosure. In the drawings:

FIG. 1 is a block diagram of a logical conception of a network;

FIG. 2 illustrates an L-shaped backplane;

FIG. 3 illustrates an L-shaped backplane from a top of a rack;

FIG. 4 illustrates an example leaf switch bar;

FIG. 5 illustrates leaf switch bars connected along correspondingconnectivity bars of an L-shaped backplane from a top of a rack;

FIG. 6 illustrates leaf switch bars plug along correspondingconnectivity bars of an L-shaped backplane;

FIG. 7 illustrates an example even Top-of-Pod (ToP) switch bar;

FIG. 8 illustrates a top view of a placement and connection of ToPswitch bars to leaf switch bars;

FIG. 9 illustrates a placement and connection of a ToP switch bar (e.g.,even type) to a plurality of leaf switch bars;

FIG. 10 illustrates a first pod, a second pod, and a third pod stackedupon one another;

FIG. 11 illustrates an example Top-of-Fabric (ToF) switch bar;

FIG. 12 illustrates ToF switch bars inserted into and connected with thestacked pods shown in FIG. 10;

FIG. 13 illustrates a top view of FIG. 8 of a placement and connectionof ToP switch bars to leaf switch bars with ToF switch bars placed infirst interstice gaps;

FIG. 14 illustrates a ringing layer used to connect ToF switch barstogether;

FIG. 15 illustrates a ringing layer used to connect ToF switch barstogether;

FIG. 16 illustrates rotating a networking device;

FIG. 17 illustrates redundant power supplies;

FIG. 18 illustrates a cooling system;

FIG. 19 illustrates a retractable mechanical connector mechanism; and

FIG. 20 illustrates an optical connector mechanism.

DETAILED DESCRIPTION

Overview

Power supply for a networking device may be provided. The networkingdevice may comprise a first plurality of switch bars each comprising afirst switch type arranged parallel to one another and a secondplurality of switch bars each comprising a second switch type arrangedparallel to one another. The first plurality of switch bars and thesecond plurality of switch bars may be arranged orthogonally. A firstplurality of power supplies may be fed by a first source. A secondplurality of power supplies may be fed by a second source. Respectiveones of a first portion of the first plurality of power supplies feedfirst respective pairs of the first plurality of switch bars andrespective ones of a first portion of the second plurality of powersupplies feed second respective pairs of the first plurality of switchbars. The first respective pairs of the first plurality of switch barsmay be different from the second respective pairs of the first pluralityof switch bars.

Both the foregoing overview and the following example embodiments areexamples and explanatory only, and should not be considered to restrictthe disclosure's scope, as described and claimed. Furthermore, featuresand/or variations may be provided in addition to those described. Forexample, embodiments of the disclosure may be directed to variousfeature combinations and sub-combinations described in the exampleembodiments.

Example Embodiments

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar elements.While embodiments of the disclosure may be described, modifications,adaptations, and other implementations are possible. For example,substitutions, additions, or modifications may be made to the elementsillustrated in the drawings, and the methods described herein may bemodified by substituting, reordering, or adding stages to the disclosedmethods. Accordingly, the following detailed description does not limitthe disclosure. Instead, the proper scope of the disclosure is definedby the appended claims.

A logical conception of a network may be illustrated in FIG. 1.Embodiments of the disclosure may comprise turning the logicalconception shown in FIG. 1 into a physical apparatus. FIG. 1 shows athree-level hierarchy of switching resources 100, as commonlyimplemented in fat pipe architectures in high capacity data centers.These levels may include a Top-of-Fabric (ToF) level 105, a Top-of-Pod(ToP) level 110, and a leaf level 115, each implemented by a unique typeof switch or router chassis. The ToP and leaf levels may be grouped intopods. Connections shown on the bottom of the pods may go to individualelements in the data center, such as servers, storage engines, ornetwork devices. Interconnects, comprising Ethernet cables, may connectthe various levels. There may be inefficiencies in conventionalimplementations, including a large number of cables, significant powerto drive, physical space to terminate them, scaling concerns, andserviceability challenges.

By building switches in the form of elongated rectangular prism shapedbars with ports (e.g., connectors) on two or more sides of a bar, athree-dimensional apparatus may be built that may implement a fat treewithout wires connecting the switches. Each of these bars may take theplace of an entire chassis-level switch described in FIG. 1. Thisapparatus may be efficient in terms of space, power, and cost. Forexample, with respect to FIG. 1, for switches with 8 ports, KLeaf=4, anapparatus with KLeaf=KTop=Nb of Pods may be provided, though KLeaf=8,12, 16, or 24 may also be used. In the FIG. 1 example, a cube of fourpods (e.g., PoD 1, PoD 2, PoD3, and PoD 4) may be provided.

Embodiments of the disclosure may have N the same for all threedimensions resulting in a cube, but there may be a different value, forexample, M, L, and N in all three dimensions. Accordingly, there may bein each PoD K bars of 2*N ports as leaves and N bars of 2*K ports asToP. There may be 2*M ports on a ToF bar, M being the maximum number ofToP that the bar can connect.

Consistent with embodiments of the disclosure, a bar (e.g., an elongatedrectangular prism shaped bar) as a highly integrated switch mayinterconnect with other similar bar shaped switches in orthogonal planesvia connection ports on its sides. A bar may be a rectangular prism onthe order of 20-40 mm wide, 20-40 mm tall, and up to 800 mm long forexample. These bars may be stacked in a three-dimensional array, withnumerous contact points where orthogonal bars touch, and variouschannels and apertures between them. As will be described in greaterdetail below, vertical bars in every other vertical hole may provide aspine (i.e., Top-of-Fabric (ToF) in Routing in Fat Trees (RIFT)). Thismay create cable-free interconnect paths between leaf switches andTop-of-Pod (ToP) switches (described in greater detail below) in thecontact points between two of the dimensions, and the paths between ToPand ToF switches in the contact points between a different twodimensions. In addition to the connection ports (e.g., connectors) onvarious long sides of the bars, there may also be inlet and exhaustapertures for coolant. Power and management links may be on one of thesmall surfaces of the bars. Together, this array of three dimensions ofswitch bars may implement a fat tree without the equipment and cablingcomplexities found in conventional data center networkingimplementations.

The links between switches (or routers) do not usually connect otherdevices. Thus, they may not have to be implemented as expensive, powerhungry, and space inefficient Cat6 RJ45 connectors and they may not haveto be implemented on the face of the switch. Embodiments of thedisclosure may include smaller ports (i.e., connectors), and may placethem on the top and the bottom of the bottom switch bars (e.g., the leafswitches) and on the bottom and the sides of the top switch bars (e.g.,the ToP switches). The vertical bars may comprise spine switches (i.e.,ToF switches), with connectors on their fronts and backs.

With embodiments of the disclosure, the vertical spine switches (i.e.,ToF switches) may have ports (i.e., connectors) on both sides, then thenumber of planes may be divided by two and provide 2X redundancy in thefabric in each plane. Accordingly, consistent with embodiments of thedisclosure:

-   -   a) connections may be established by stacking switch bars in        three different orientations;    -   b) inter-switch cables may be replaced with simpler bar contact        point connections; and    -   c) connections may be smaller, thus equipment density may be        higher, and overall power dissipation reduced.

Consistent with embodiments of the disclosure, pods may include a firstplurality of switch bars (e.g., leaf switch bars) oriented, for example,parallel to the X axis, and a second plurality of switch bars (e.g., ToPswitch bars) directly above them parallel to the Y axis, creating a pod.A plurality of these two layered pods may be stacked in a rack, and athird plurality of switch bars (e.g., spine switch bars or ToF switchbars) may be threaded through a subset of gaps left between the firstplurality of switch bars and the second plurality of switch bars,parallel to the Z axis, thus completing three mutually orthogonal planesof switching for a fat tree architecture.

Building a Pod

The leaf switches (e.g., first plurality of switch bars) may comprisebars, with 2×N ports, half of the ports on the top of the bar and halfon the bottom of the bars. The bottom may connect to host servers. Inorder to bring the connectivity from the bottoms of the leaf switch barsto external cables to the host servers, an “L-shaped” backplane 200 maybe used. As shown in FIG. 2, L-shaped backplane 200 may comprise aplurality of connectivity bars 205 and a front plate 210 having aplurality of front plate connectors 215. L-shaped backplane 200 mayconnect the ports coming out of the array of leaf switches (e.g., firstplurality of switch bars) to a connector panel (i.e., front plate 210)where, for example, RJ-45 metallic connectors or fiber interface modules(i.e., front plate connectors 215) may be located to connect to hostservers. Although one front plate 210 serving an entire pod is shown inFIG. 2, embodiments of the disclosure may have one smaller panel foreach leaf switch bar, individually removable and replaceable forexample.

The connectors on front plate 210 (i.e., front plate connectors 215) maycomprise, but are not limited to, RJ-45 connectors or pluggable modulessuch as Small Form-factor Pluggable (SFP), Quad Small Form-factorPluggable (QSFP), 10 gigabit Small Form-factor Pluggable (XFP), or othertypes. FIG. 3 shows L-shaped backplane 200 from a top of a rack.Connectors between L-shaped backplane 200 and the first layer of switchbars may not necessarily be RJ-45 and may be designed so that the femaleon the top side may connect to a male on the bottom side while the podswitch above is oriented at 90 degrees. To enable an easy insertion, theconnectors may be point shaped or optical. As described herein, theconnectors may be illustrated in the shape of an eight-pointed star forexample.

FIG. 4 illustrates an example leaf switch bar 400. As shown in FIG. 4,leaf switch bar 400 may comprise a top 405 comprising a plurality ofconnectors, a bottom 410 comprising a plurality of connectors, a firstside 415, and a second side 420. The cross-sectional size of leaf switchbars 400 may be based on the number of bars placed across the intendedmounting rack. One connector per leaf switch bar 400 output port (e.g.,serving cables connecting to servers, storage devices, networkingdevices, or other equipment in a wire center) may appear on bottom 410,and one connector per ToP switch may appear on top 405. A first end 425of leaf switch bar 400 may include a set of power and management portconnectors. A second end 430 of leaf switch bar 400 may include statusindicator LEDs, and retention latches. Also, first side 415 and secondside 420 of leaf switch bar 400 may include some apertures for coolantto pass through leaf switch bar 400 and across internal heat sinks asdescribed in greater detail below. As shown in FIG. 5 and FIG. 6, leafswitch bars 400 may plug along corresponding connectivity bars 205 ofL-shaped backplane 200.

Top of Pod (ToP) Switch Level

The switch level above in the leaf switches (i.e., the first pluralityof switch bars) may comprise the ToP switches (i.e., the secondplurality of switch bars), which may be formed from similar bars at theleaf switches, but rotated 90 degrees. A cross layer of bars may beapplied by plugging bars on bars so that every bar at the lower layer(i.e., leaf switches) may be connected to every bar at the above layer(i.e., ToP switches). This may provide NΛ2 contact points, whichimplement the leaf to ToP switch cables shown in FIG. 1. The ToP switchbars may be similar to the leaf switch bars, but the connectors may bearranged differently.

FIG. 7 illustrates an example ToP switch bar 700. As shown in FIG. 7,ToP switch bar 700 may have a first side 705, a top 710, a second side715, and a bottom 720. Bottom 720 of ToP switch bar 700 may have aplurality of connectors to interconnect with leaf switch bars 400arranged in an orthogonal direction directly below. Second side 715 ofeach ToP switch bar 700 may have a plurality of connectors that connectto the ToF switch bars (i.e., spine switch bars or the third pluralityof switch bars) that may be immediately adjacent, but arrangedvertically. As before, one end 725 of the ToP switch bar 700 may haveconnectors to provide power and management, and another end 730 mayinclude status indicators and retention latches. In some embodiments,cooling passages may exist on first side 705 and second side 715 toallow passage of coolant between cooling channels through heat-producingelements inside ToP switch bar 700.

Consistent with embodiments of the disclosure, the number of planes maybe divided where even and odd type ToP switch bars 700 may be used. Thismay allow ToF switch bars (i.e., spine switch bars or the thirdplurality of switch bars) in the middle to connect to ToP switch bars700 both towards the back and the front of the rack. FIG. 7 illustratean even type ToP switch bar 700. With an odd type ToP switch bar 700,first side 705 may have the plurality connectors that connect to the ToFswitch bars (i.e., spine switch bars or the third plurality of switchbars) rather than second side 715.

FIG. 8 illustrates a top view of the placement and connection of ToPswitch bars 700 to leaf switch bars 400 to form a pod. As shown in FIG.8, the ToF switch bars may be placed in first interstice gaps 805 andsecond interstice gaps 810 may be used for cooling passages as describedin greater detail below. The contact surfaces of even and odd ToP switchbars 700 face each other across first interstice gaps 805. FIG. 9illustrates the placement and connection of ToP switch bar 700 (e.g.,even type) to a plurality of leaf switch bars 400.

Multiple Pods

One objective of a Clos or a fat tree may be to have many podsinterconnected by a spine switch (i.e., ToF switch). Each Pod maycomprise one or more Rack Units (RUs) high. Multiple pods may be placedin a rack, one above the other all the way up the rack. FIG. 10illustrates a first pod 1005, a second pod 1010, and a third pod 1015stacked upon one another. In the cross-section shown in FIG. 10,connectors (e.g., the eight-pointed stars) on the sides of odd ToPswitches 700 may be connected to the vertical ToF switch bars thatinterlink pods as described in greater detail below.

The Spine

FIG. 11 illustrates an example ToF switch bar 1100. ToF switch bar 1100may be used for spine switch bars (i.e., the third plurality of switchbars). As shown in FIG. 11, ToF switch bar 1100 may have a first side1105, a top 1110, a second side 1115, and a bottom 1120. First side 1105of ToF switch bar 1100 may have a plurality of connectors tointerconnect with ToP switch bars 700 (e.g., even type) arranged in anorthogonal direction. Second side 1115 of each ToF switch bar 1100 mayhave a plurality connectors that connect to ToP switch bars 700 (e.g.,odd type) arranged in an orthogonal direction.

ToF switch bars 1100 may be organized vertically. Because they haveports on both sides, the number of planes may be divided by 2, meaningK/2 planes total (because each ToF switch bar 1100 may be able toservice two rows of ToP switch bars 700 in each pod). There may beconnectors on ends 1125 of ToF switch bars 1100 that may connect powerand management to ToF switch bars 1100, and also for expansionconnections (i.e., communications connector) to a plurality of ringingbars to provide a ringing layer as described in greater detail below.

Connectors on ends 1125 of ToF switch bars 1100 may also be used toconnect ToF switch bars 1100 together end-to-end to lengthen ToF switchbars 1100 if the original is, for example, half the full size. This mayallow the ToF level to be expanded as needed, for example, starting witha quarter or one-half of the maximum number of pods and then addingmore. Initially, it may be enough that ToF switch bars 1100 are aquarter or one-half the number of ports, as long as they can be expandedby plugging in additional ToF switch bars 1100 end-to-end.

FIG. 12 illustrates ToF switch bars 1100 inserted into and connectedwith the stacked pods shown in FIG. 10. The size of the structure shownin FIG. 12 may be increased by expanding ToF switch bars 1100 asdescribed above and adding more pods. FIG. 13 illustrates the top viewof FIG. 8 of the placement and connection of ToP switch bars 700 to leafswitch bars 400. In contrast with FIG. 8, FIG. 13 shows ToF switch bars1100 placed in first interstice gaps 805, making connections betweentheir side 1105 connectors and odd ToP switch bar side 715 connectors.Second interstice gaps 810 may be empty in FIG. 13, but may be used forcoolant as described in greater detail below.

FIG. 14 and FIG. 15 illustrate a ringing layer that may be used toconnect ToF switch bars 1100 together. Embodiments of the disclosure mayprovide an optional ringing layer above ToF switch bars 1100 using aplurality of ringing bars 1400 to provide the ringing layer. Ringingbars 1400 may be similar to leaf switch bars 400. FIG. 14 is a top viewshowing a first plurality of ringing bars 1405 connected to ends 1125 ofToF switch bars 1100 of FIG. 13. FIG. 15 shows a second plurality ofringing bars 1505 arranged orthogonally to and connected to firstplurality of ringing bars 1405 of FIG. 14. Accordingly, the combinationof first plurality of ringing bars 1405 and second plurality of ringingbars 1505 may connect ToF switch bars 1100 together and thus provide aringing layer. Second plurality of ringing bars 1505 shown in FIG. 15may have their second side of ports on the top that may to provideconnectivity to management servers, or to routers that route to othersimilar three-dimensional switches.

Serviceability

A networking device comprising the aforementioned leaf switch bars 400,ToP switch bars 700, and ToF switch bars 1100 may be assembled andconnected as described above and placed in a standard rack. However, ifthe networking device is mounted in a standard rack in a conventionalway, serviceability may be a problem. For example, while one type ofswitch bars (e.g., switch leaf switch bars 400) may be replaceable bysliding them from the networking device through the front of the rack,the orthogonal dimension (e.g., ToP switch bars 700) may not beserviceable because they would have to be withdrawn through themechanical uprights on the side of the rack or perhaps even requireadditional service volume in the space occupied by adjacent racks.

FIG. 16 illustrates a way to address the serviceability problem byrotating a networking device 1600 about the Z axis by approximately 45degrees (or any angle that may facilitate improvements in serviceabilityand equipment density). As shown in FIG. 16, networking device 1600 maystay within the dashed outline of a standard rack. This may allow bothleaf switch bars 400 and ToP switch bars 700 to be slidably removedthrough the front of the rack without interference from the rack'sstructural components (e.g., a first upright 1605 and a second upright1610) or adjacent racks, as shown by the dotted positions of thepartially withdrawn leaf switch bar 400 and ToP switch bar 700.

This rotation may be fixed, or networking device 1600 may be mounted onhinges (or a hinged device) allowing it to pivot about a pivot point1615 to the position shown in FIG. 16 for servicing, and pivoted back toa position more fully between first upright 1605 and second upright 1610for normal operation. ToF switch bars 1100 may also have serviceabilitychallenges that may be addressed by leaving space above networkingdevice 1600 in the rack open, and pulling ToF switch bars 1100 outthrough the top of the rack. In other embodiments, a second axis ofpivot may allow networking device 1600 to tip down to facilitate thewithdrawal of ToF switch bars 1100. Also, as stated above, the ToFswitch bar 1100 length may be divided (e.g., by 2 or 4). Accordingly,ToF switch bars 1100 may be disassembled as they are withdrawn from thetop of networking device 1600, and reassemble as replacement parts arethey are inserted. This may use a shorter mechanical volume, which mayfit the space available in the rack.

Redundant Power Supplies

Consistent with embodiments of the disclosure, redundant power suppliesmay be provided to networking device 1600. Each pod may have an array ofpower supplies that may comprise power conditioning and distributionelements. For example, as shown in FIG. 17, the array of power suppliesmay comprise a first plurality of power supplies 1705 fed by a firstsource and a second plurality of power supplies 1710 fed by a secondsource.

Power supplies alternating between first plurality of power supplies1705 and second plurality of power supplies 1710 may be shared amongpairs of leaf switch bars 400. In other words, each one of leaf switchbars 400 may be fed from the first source through one of first pluralityof power supplies 1705 and each one of leaf switch bars 400 may be fedfrom the second source through one of second plurality of power supplies1710. This may provide fault tolerance should any single power supplyfail or if one of the first source or the second source is lost.

Similarly, power supplies alternating between first plurality of powersupplies 1705 and second plurality of power supplies 1710 may be sharedamong pairs of ToP switch bars 700. In other words, each one of ToPswitch bars 700 may be fed from the first source through one of firstplurality of power supplies 1705 and each one of ToP switch bars 700 maybe fed from the second source through one of second plurality of powersupplies 1710.

Furthermore, ToF switch bars 1100 may be fed by first plurality of powersupplies 1705 and second plurality of power supplies 1710 in a waysimilar to how leaf switch bars 400 and ToP switch bars 700 are fed asdescribed above, but these power supplies may be located belownetworking device 1600. In this way, because the power supplies ofnetworking device 1600 may be fed from multiple sources, power may notbe lost to networking device 1600 when one of the sources is lost. Also,power may not be lost to a switch bar when a power supply fails.

While FIG. 17 illustrates one pod level of networking device 1600, theredundant power supply system described above may be repeated for otherpod levels of networking device 1600. For example, each pod level ofnetworking device 1600 may have its own first plurality of powersupplies 1705 and second plurality of power supplies 1710 or several podlevels of networking device 1600 may share first plurality of powersupplies 1705 and second plurality of power supplies 1710. In otherwords, power supplies may serve switch bars of only a single pod level,or may extend vertically up the rack to service switch bars in multiple(or all) pod levels. Furthermore, when leaf switch bars 400, ToP switchbars 700, or ToF switch bars 1100 are slidably inserted into networkingdevice 1600 or slidably removed from networking device 1600, switch barsmay make a blind connection or disconnection from their respectiveredundant power supplies.

Cooling

FIG. 18 illustrates a cooling system consistent with embodiments of thedisclosure. Due to the densely packed architecture of networking device1600, cooling may be needed. As described above with respect to FIG. 8and FIG. 13, second interstice gaps 810 may be used for coolingpassages. As shown in FIG. 18, networking device 1600 may include aplurality of cooling passages (i.e., second interstice gaps 810). Forexample, half of the passages (i.e., first interstice gaps 805 of FIG.8) between leaf switch bars 400 and ToP switch bars 700 may be filled byToF switch bars 1100, leaving the other half (i.e., second intersticegaps 810 of FIG. 8) to support cooling as cooling passages.

Cooling flows may be divided into two sets of vertical passages passingup through the pods in networking device 1600. A first set of verticalpassages may comprise a first portion of the plurality of the coolingpassages and may supply coolant. This first set of vertical passages maybe pressurized by fans below the bottom pod of networking device 1600and may be blocked on top of the top pod of networking device 1600. Asecond set of the vertical passages may comprise a second portion of theplurality of the cooling passages and may exhaust coolant. This secondset of vertical passages may be evacuated by fans above the top pod ofnetworking device 1600 and blocked from below the bottom pod ofnetworking device 1600.

Lateral channels may pass from the cooling supply passages, through oneor more switch bar, and back to cooling exhaust passages. The smallarrows show coolant flow paths through leaf switch bars 400. Thickerarrows show paths from a coolant supply channel, through one ToP switchbar 700, through a vertical ToF switch bar 1100, through a second ToPswitch bar 700 and out a cooling exhaust passage. This flow pattern maybe modified to put fewer bars in series to improve the thermalperformance. Baffles lining both the cooling supply passages and thecooling exhaust passages may, for example, direct required amounts ofcoolant through ducts and through the switch bars, where internal heatsinks dissipate the heat generated by the switch bar's internalcomponents, and carry it away with the flowing coolant.

In another embodiment, the aforementioned coolant supply and exhaustpassages may be replaced with liquid cooled bars disposed in secondinterstice gaps 810. These liquid cooled bars may have cold liquidcoolant flowing from below the bottom pod of networking device 1600,through internal channels in the liquid cooled bars chilling theirsides, up to the top, turning 180 degrees, and back down, and the warmedcoolant may be collected below the bottom pod of networking device 1600.Two such coolant paths may be provided in each liquid cooled bar, andisolated from each other to provide cooling redundancy. Fluid manifoldsbelow the bottom pod may connect the liquid cooled bars to coolantsupply and return plumbing. Each switch bar may have heat pipes andthermal interfaces (similar to “wedge locks”) to thermally connect thehigh power components within the switch bar to the surface of the liquidcooled bars to remove the excess heat to the flowing liquid coolant. Theheat from dissipative components within leaf switch bars 400, ToP switchbars 700, and vertical ToF switch bars 1100 may be moved to the liquidcooled bars. This may be accomplished through mechanical contact betweenselected points (that may be thermally tied to hot components within theswitch bars) on the outside of the switch bars and cold bars.

Connectors

FIG. 19 and FIG. 20 illustrate connectors that may be used to connectconnectivity bars 205, leaf switch bars 400, ToP switch bars 700, ToFswitch bars 1100, and ringing bars 1400 consistent with embodiments ofthe disclosure. Connectors may be illustrated, for example, byeight-pointed stars in FIGS. 2 through 7 and FIGS. 9 through 14. Theconnectors may be inexpensive, compact, reliable, and have adequateperformance to pass bidirectional data links in, for example, the 10Gb/s-40 Gb/s range across a few centimeters between circuit boards inadjacent switch bars. The connectors may support the serviceabilityscenarios where any switch bar is slidably removed from networkingdevice 1600 or slidably inserted into networking device 1600 withoutinterfering with or damaging the connectors from adjacent or orthogonalswitch bars. Embodiments of the disclosure may include a retractablemechanical connector mechanism (e.g., FIG. 19) and an optical connectormechanism (e.g., FIG. 20).

FIG. 19 illustrates a retractable mechanical connector mechanism 1900.Retractable mechanical connector mechanism 1900 may comprise an ejectorhandle 1905 having a cam 1910, a linkage 1915 activated by cam 1910, atleast one pin (e.g., one of plurality of pins 1920) having a head (e.g.,one of plurality of heads 1925) connected to linkage 1915, and acompression element 1930 adjacent to the head. Plurality of pins 1920may be retractable by operation of retractable mechanical connectormechanism 1900 to cover, for example, a few centimeters of distancebetween circuit boards inside of a pair of switch bars. An array oftelescoping pins (e.g., plurality of pins 1920 that may be similar to“pogo pins” used in circuit board test fixtures) may extend beyond theboundaries of a switch bar or be retracted within the switch bar if thatswitch bar is to be removed from or inserted into networking device1600.

Retractable mechanical connector mechanism 1900 shown in FIG. 19 may beactivated by ejector handle 1905 that may be operated when leaf switchbar 400 is to be removed from or inserted into networking device 1600.When ejector handle 1905 is operated to its retraction position (e.g.,shown in the dotted line position), cam 1910 may push down linkage 1915that may move down “T” shaped plurality of heads 1925 at the ends ofeach of plurality of pins 1920. This operation of ejector handle 1905may compress compression element 1930 (e.g., comprising, for example, anelastomer spring) to move plurality of pins 1920 down fully inside anenclosure of leaf switch bar 400. This may create enough clearance forleaf switch bar 400 to slidably move without plurality of pins 1920grinding on ToP switch bar 700 as it passes.

After leaf switch bar 400 is inserted into networking device 1600,ejector handle 1905 may be returned to its seated (e.g., fully inserted)position, linkage 1915 may move back up, and compression element 1930may gently extend plurality of pins 1920 until they make solid contactwith corresponding pins (i.e., a plurality of pins 1935) coming from ToPswitch bar 700. ToP switch bar 700 may have a retractable mechanicalconnector mechanism similar to retractable mechanical connectormechanism 1900 to retract its plurality of pins 1935 up and inside ToPswitch bar 700 so that ToP switch bar 700 may be slidably moved withoutplurality of pins 1935 colliding with plurality of pins 1920 of leafswitch bars 400 as they pass by. Although four pins are shown inplurality of pins 1920 and plurality of pins 1935, embodiments of thedisclosure may include any number of pins and is not limited to fourpins. Connections between connectivity bars 205 and leaf switch bars400, ToP switch bars 700 and ToF switch bars 1100, and ToF switch bars1100 and ringing bars 1400 may use a similar connector mechanism. Thepin geometry, spacing, and the dialectic materials between plurality ofpins 1920 and plurality of pins 1935 may be chosen to provide thedesired impedance or transmission line characteristics.

FIG. 20 illustrates an optical connector mechanism 2000 consistent withembodiments of the disclosure. As shown in FIG. 20, instead of mechanicsand linkages to move physical electrical contacts, this embodiment mayuse free space optical light beams to cross the few centimeters betweenswitch bars. Optical connector mechanism 2000 may begin with a TransmitOptical Sub Assembly (TOSA) that may accept serial bit streams fromwithin leaf switch bar 400 and convert them to modulated opticalsignals. A lens may shape and collimate the emitted light that crosses agap between leaf switch bar 400 and ToP switch bar 700. Another lens inan optical connector mechanism 2005 of ToP switch bar 700 may interceptthe beam and may focus it on a Receive Optical Sub Assembly (ROSA) thatmay demodulate and create an electrical signal for ToP switch bar 700.Pairs of receivers and transmitters may be used back-to-back to createbidirectional links. Multiple links may be run in parallel to increasethe capacity. Because there may be no protruding pins to interfere withswitch bar movements, this embodiment may be more mechanically robust.However, the optical embodiment may be more expensive and may consumemore power.

Embodiments of the disclosure may do away with thousands of inter-switchcables. It may improve the volumetric density of the switching solutionsfor highly interconnected data centers, freeing up more rack space forservers. It may be easier to install, monitor, and service. It mayenable lower power dissipation for a data center's switching functions.It also may have important pay-as-you-grow properties, so it may beunnecessary to install full complements of full-sized switches from theonset. Individual bars and pods may be equipped as necessary as servercounts increase and traffic grows.

Embodiments of the disclosure may comprise a system. The system maycomprise a first plurality of switch bars each comprising a first switchtype arranged parallel to one another; a second plurality of switch barseach comprising a second switch type arranged parallel to one another; athird plurality of switch bars each comprising a third switch typearranged parallel to one another, wherein the first plurality of switchbars, the second plurality of switch bars, and the third plurality ofswitch bars are arranged orthogonally; a first plurality of powersupplies being fed by a first source; and a second plurality of powersupplies being fed by a second source wherein respective ones of a firstportion of the first plurality of power supplies feed first respectivepairs of the first plurality of switch bars and respective ones of afirst portion of the second plurality of power supplies feed secondrespective pairs of the first plurality of switch bars wherein the firstrespective pairs of the first plurality of switch bars are differentfrom the second respective pairs of the first plurality of switch barswherein power is fed to each of the first plurality of switch bars fromboth the first source and the second source.

The first switch type may comprise a leaf switch, the second switch typemay comprise a top of pod switch, and the third switch type may comprisea top of fabric switch.

The first plurality of switch bars may be adjacent to and connected tothe second plurality of switch bars and wherein the second plurality ofswitch bars are adjacent to and connected to the third plurality ofswitch bars.

The first plurality of switch bars may be connected to the secondplurality of switch bars via optical connectors and wherein the secondplurality of switch bars are connected to the third plurality of switchbars via optical connectors.

The first plurality of switch bars may be connected to the secondplurality of switch bars via mechanical connectors and wherein thesecond plurality of switch bars are connected to the third plurality ofswitch bars via mechanical connectors.

Respective ones of a second portion of the first plurality of powersupplies feed first respective pairs of the second plurality of switchbars and respective ones of a second portion of the second plurality ofpower supplies feed second respective pairs of the second plurality ofswitch bars wherein the first respective pairs of the first plurality ofswitch bars are different from the second respective pairs of the firstplurality of switch bars wherein power is fed to each of the secondplurality of switch bars from both the first source and the secondsource.

Respective ones of a third portion of the first plurality of powersupplies feed first respective pairs of the third plurality of switchbars and respective ones of a third portion of the second plurality ofpower supplies feed second respective pairs of the third plurality ofswitch bars wherein the first respective pairs of the third plurality ofswitch bars are different from the second respective pairs of the thirdplurality of switch bars wherein power is fed to each of the thirdplurality of switch bars from both the first source and the secondsource.

The first plurality of power supplies may comprise power conditioning,protection, and distribution elements.

The second plurality of power supplies comprise power conditioning,protection, and distribution elements.

Each one of the first plurality of switch bars, the second plurality ofswitch bars, and the third plurality of switch bars are configured to beslidably removed from the system and slidably inserted into the system.

Embodiments of the disclosure may comprise a system. The system maycomprise a first plurality of switch bars each comprising a first switchtype arranged parallel to one another; a second plurality of switch barseach comprising a second switch type arranged parallel to one another,wherein the first plurality of switch bars and the second plurality ofswitch bars are arranged orthogonally; a first plurality of powersupplies being fed by a first source; and a second plurality of powersupplies being fed by a second source wherein respective ones of a firstportion of the first plurality of power supplies feed first respectivepairs of the first plurality of switch bars and respective ones of afirst portion of the second plurality of power supplies feed secondrespective pairs of the first plurality of switch bars wherein the firstrespective pairs of the first plurality of switch bars are differentfrom the second respective pairs of the first plurality of switch barswherein power is fed to each of the first plurality of switch bars fromboth the first source and the second source.

The first switch type may comprise a leaf switch and the second switchtype may comprise a top of pod switch.

Respective ones of a second portion of the first plurality of powersupplies feed first respective pairs of the second plurality of switchbars and respective ones of a second portion of the second plurality ofpower supplies feed second respective pairs of the second plurality ofswitch bars wherein the first respective pairs of the first plurality ofswitch bars are different from the second respective pairs of the firstplurality of switch bars wherein power is fed to each of the secondplurality of switch bars from both the first source and the secondsource.

The system may further comprise a third plurality of switch bars eachcomprising a third switch type arranged parallel to one another, whereinthe first plurality of switch bars, the second plurality of switch bars,and the third plurality of switch bars are arranged orthogonally.

The third switch type comprises a top of fabric switch.

Embodiments of the disclosure may comprise a method. The method maycomprise providing a first plurality of switch bars each comprising afirst switch type arranged parallel to one another; providing a secondplurality of switch bars each comprising a second switch type arrangedparallel to one another; providing a third plurality of switch bars eachcomprising a third switch type arranged parallel to one another, whereinthe first plurality of switch bars, the second plurality of switch bars,and the third plurality of switch bars are arranged orthogonally;providing a first plurality of power supplies being fed by a firstsource; and providing a second plurality of power supplies being fed bya second source wherein respective ones of a first portion of the firstplurality of power supplies feed first respective pairs of the firstplurality of switch bars and respective ones of a first portion of thesecond plurality of power supplies feed second respective pairs of thefirst plurality of switch bars wherein the first respective pairs of thefirst plurality of switch bars are different from the second respectivepairs of the first plurality of switch bars wherein power is fed to eachof the first plurality of switch bars from both the first source and thesecond source.

Providing the first switch type may comprise providing the first switchtype comprising a leaf switch; providing the second switch type maycomprise providing the second switch type comprising a top of podswitch; and providing the third switch type may comprise providing thethird switch type comprising a top of fabric switch.

Embodiments of the disclosure, for example, may be implemented as acomputer process (method), a computing system, or as an article ofmanufacture, such as a computer program product or computer readablemedia. The computer program product may be a computer storage mediareadable by a computer system and encoding a computer program ofinstructions for executing a computer process. The computer programproduct may also be a propagated signal on a carrier readable by acomputing system and encoding a computer program of instructions forexecuting a computer process. Accordingly, the present disclosure may beembodied in hardware and/or in software (including firmware, residentsoftware, micro-code, etc.). In other words, embodiments of the presentdisclosure may take the form of a computer program product on acomputer-usable or computer-readable storage medium havingcomputer-usable or computer-readable program code embodied in the mediumfor use by or in connection with an instruction execution system. Acomputer-usable or computer-readable medium may be any medium that cancontain, store, communicate, propagate, or transport the program for useby or in connection with the instruction execution system, apparatus, ordevice.

The computer-usable or computer-readable medium may be, for example, butnot limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, device, or propagationmedium. More specific computer-readable medium examples (anon-exhaustive list), the computer-readable medium may include thefollowing: an electrical connection having one or more wires, a portablecomputer diskette, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, and a portable compact disc read-only memory(CD-ROM). Note that the computer-usable or computer-readable mediumcould even be paper or another suitable medium upon which the program isprinted, as the program can be electronically captured, via, forinstance, optical scanning of the paper or other medium, then compiled,interpreted, or otherwise processed in a suitable manner, if necessary,and then stored in a computer memory.

While certain embodiments of the disclosure have been described, otherembodiments may exist. Furthermore, although embodiments of the presentdisclosure have been described as being associated with data stored inmemory and other storage mediums, data can also be stored on or readfrom other types of computer-readable media, such as secondary storagedevices, like hard disks, floppy disks, or a CD-ROM, a carrier wave fromthe Internet, or other forms of RAM or ROM. Moreover, the semantic dataconsistent with embodiments of the disclosure may be analyzed withoutbeing stored. In this case, in-line data mining techniques may be usedas data traffic passes through, for example, a caching server or networkrouter. Further, the disclosed methods' stages may be modified in anymanner, including by reordering stages and/or inserting or deletingstages, without departing from the disclosure.

Furthermore, embodiments of the disclosure may be practiced in anelectrical circuit comprising discrete electronic elements, packaged orintegrated electronic chips containing logic gates, a circuit utilizinga microprocessor, or on a single chip containing electronic elements ormicroprocessors. Embodiments of the disclosure may also be practicedusing other technologies capable of performing logical operations suchas, for example, AND, OR, and NOT, including, but not limited to,mechanical, optical, fluidic, and quantum technologies. In addition,embodiments of the disclosure may be practiced within a general purposecomputer or in any other circuits or systems.

Embodiments of the disclosure may be practiced via a system-on-a-chip(SOC) where each or many of the components may be integrated onto asingle integrated circuit. Such an SOC device may include one or moreprocessing units, graphics units, communications units, switchingfabrics, routers, system virtualization units and various applicationfunctionality of which may be integrated (or “burned”) onto the chipsubstrate as a single integrated circuit. When operating via an SOC, thefunctionality described herein with respect to embodiments of thedisclosure, may be performed via application-specific logic integratedwith other components on the single integrated circuit (chip).

Embodiments of the present disclosure, for example, are described abovewith reference to block diagrams and/or operational illustrations ofmethods, systems, and computer program products according to embodimentsof the disclosure. The functions/acts noted in the blocks may occur outof the order as shown in any flowchart. For example, two blocks shown insuccession may in fact be executed substantially concurrently or theblocks may sometimes be executed in the reverse order, depending uponthe functionality/acts involved.

While the specification includes examples, the disclosure's scope isindicated by the following claims. Furthermore, while the specificationhas been described in language specific to structural features and/ormethodological acts, the claims are not limited to the features or actsdescribed above. Rather, the specific features and acts described aboveare disclosed as example for embodiments of the disclosure.

What is claimed is:
 1. A system comprising: a first plurality of switchbars each comprising a first switch type arranged parallel to oneanother; a second plurality of switch bars each comprising a secondswitch type arranged parallel to one another; a third plurality ofswitch bars each comprising a third switch type arranged parallel to oneanother, wherein the first plurality of switch bars, the secondplurality of switch bars, and the third plurality of switch bars arearranged orthogonally; a first plurality of power supplies being fed bya first source; and a second plurality of power supplies being fed by asecond source wherein respective ones of a first portion of the firstplurality of power supplies feed first respective pairs of the firstplurality of switch bars and respective ones of a first portion of thesecond plurality of power supplies feed second respective pairs of thefirst plurality of switch bars wherein the first respective pairs of thefirst plurality of switch bars are different from the second respectivepairs of the first plurality of switch bars wherein power is fed to eachof the first plurality of switch bars from both the first source and thesecond source.
 2. The system of claim 1, wherein the first switch typecomprises a leaf switch.
 3. The system of claim 1, wherein the secondswitch type comprises a top of pod switch.
 4. The system of claim 1,wherein the third switch type comprises a top of fabric switch.
 5. Thesystem of claim 1, wherein the first plurality of switch bars areadjacent to and connected to the second plurality of switch bars andwherein the second plurality of switch bars are adjacent to andconnected to the third plurality of switch bars.
 6. The system of claim5, wherein the first plurality of switch bars are connected to thesecond plurality of switch bars via optical connectors and wherein thesecond plurality of switch bars are connected to the third plurality ofswitch bars via optical connectors.
 7. The system of claim 5, whereinthe first plurality of switch bars are connected to the second pluralityof switch bars via mechanical connectors and wherein the secondplurality of switch bars are connected to the third plurality of switchbars via mechanical connectors.
 8. The system of claim 1, whereinrespective ones of a second portion of the first plurality of powersupplies feed first respective pairs of the second plurality of switchbars and respective ones of a second portion of the second plurality ofpower supplies feed second respective pairs of the second plurality ofswitch bars wherein the first respective pairs of the first plurality ofswitch bars are different from the second respective pairs of the firstplurality of switch bars wherein power is fed to each of the secondplurality of switch bars from both the first source and the secondsource.
 9. The system of claim 1, wherein respective ones of a thirdportion of the first plurality of power supplies feed first respectivepairs of the third plurality of switch bars and respective ones of athird portion of the second plurality of power supplies feed secondrespective pairs of the third plurality of switch bars wherein the firstrespective pairs of the third plurality of switch bars are differentfrom the second respective pairs of the third plurality of switch barswherein power is fed to each of the third plurality of switch bars fromboth the first source and the second source.
 10. The system of claim 1,wherein the first plurality of power supplies comprise powerconditioning, protection, and distribution elements.
 11. The system ofclaim 1, wherein the second plurality of power supplies comprise powerconditioning, protection, and distribution elements.
 12. The system ofclaim 1, wherein each one of the first plurality of switch bars, thesecond plurality of switch bars, and the third plurality of switch barsare configured to be slidably removed from the system and slidablyinserted into the system.
 13. A system comprising: a first plurality ofswitch bars each comprising a first switch type arranged parallel to oneanother; a second plurality of switch bars each comprising a secondswitch type arranged parallel to one another, wherein the firstplurality of switch bars and the second plurality of switch bars arearranged orthogonally; a first plurality of power supplies being fed bya first source; and a second plurality of power supplies being fed by asecond source wherein respective ones of a first portion of the firstplurality of power supplies feed first respective pairs of the firstplurality of switch bars and respective ones of a first portion of thesecond plurality of power supplies feed second respective pairs of thefirst plurality of switch bars wherein the first respective pairs of thefirst plurality of switch bars are different from the second respectivepairs of the first plurality of switch bars wherein power is fed to eachof the first plurality of switch bars from both the first source and thesecond source.
 14. The system of claim 13, wherein the first switch typecomprises a leaf switch.
 15. The system of claim 13, wherein the secondswitch type comprises a top of pod switch.
 16. The system of claim 13,wherein respective ones of a second portion of the first plurality ofpower supplies feed first respective pairs of the second plurality ofswitch bars and respective ones of a second portion of the secondplurality of power supplies feed second respective pairs of the secondplurality of switch bars wherein the first respective pairs of the firstplurality of switch bars are different from the second respective pairsof the first plurality of switch bars wherein power is fed to each ofthe second plurality of switch bars from both the first source and thesecond source.
 17. The system of claim 13, further comprising a thirdplurality of switch bars each comprising a third switch type arrangedparallel to one another, wherein the first plurality of switch bars, thesecond plurality of switch bars, and the third plurality of switch barsare arranged orthogonally.
 18. The system of claim 17, wherein the thirdswitch type comprises a top of fabric switch.
 19. A method comprising:providing a first plurality of switch bars each comprising a firstswitch type arranged parallel to one another; providing a secondplurality of switch bars each comprising a second switch type arrangedparallel to one another; providing a third plurality of switch bars eachcomprising a third switch type arranged parallel to one another, whereinthe first plurality of switch bars, the second plurality of switch bars,and the third plurality of switch bars are arranged orthogonally;providing a first plurality of power supplies being fed by a firstsource; and providing a second plurality of power supplies being fed bya second source wherein respective ones of a first portion of the firstplurality of power supplies feed first respective pairs of the firstplurality of switch bars and respective ones of a first portion of thesecond plurality of power supplies feed second respective pairs of thefirst plurality of switch bars wherein the first respective pairs of thefirst plurality of switch bars are different from the second respectivepairs of the first plurality of switch bars wherein power is fed to eachof the first plurality of switch bars from both the first source and thesecond source.
 20. The method of claim 19, wherein: providing the firstswitch type comprises providing the first switch type comprising a leafswitch; providing the second switch type comprises providing the secondswitch type comprising a top of pod switch; and providing the thirdswitch type comprises providing the third switch type comprising a topof fabric switch.